build(toolchain): bump to 2022-10-16 #5119

xxchan · 2022-09-05T23:24:14Z

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

src/common/src/error.rs

xxchan · 2022-09-13T08:54:14Z

The ICE is fixed. Now we have TAIT bug 🤪 rust-lang/rust#101750

BugenZhao · 2022-10-08T05:51:56Z

Really want rust-lang/cargo#11114 after bumping. 🤤

src/storage/src/lib.rs

BugenZhao · 2022-10-16T22:27:43Z

src/meta/src/lib.rs

+#![expect(clippy::iter_kv_map, reason = "FIXME: fix later")]
+#![expect(
+    clippy::or_fun_call,


BugenZhao · 2022-10-16T22:28:12Z

src/common/src/types/chrono_wrapper.rs

-                .ok_or(ValueEncodingError::InvalidNaiveTimeEncoding(secs, nano))?,
+                .ok_or_else(|| ValueEncodingError::InvalidNaiveTimeEncoding(secs, nano))?,


There shouldn't be any difference for enum construction.

rust-lang/rust-clippy#9663

skyzh · 2022-10-17T00:37:06Z

Build of risingwave_stream will stuck on macOS. Still investigating why.

skyzh · 2022-10-17T00:41:40Z

INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::mir::ConstantKind>: result=Ok(Val(ZeroSized, for<'a, 'b, 'c> fn(std::pin::Pin<&'a mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>>, &'b mut std::task::Context<'c>) -> std::task::Poll<std::option::Option<<futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]> as tokio_stream::Stream>::Item>> {<futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]> as tokio_stream::Stream>::poll_next})) with 4052 obligations
INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::ty::subst::GenericArg>: result=Ok(unsafe fn(&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>) -> std::pin::Pin<&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>> {std::pin::Pin::<&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>>::new_unchecked}) with 4052 obligations
INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::mir::ConstantKind>: result=Ok(Val(ZeroSized, unsafe fn(&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>) -> std::pin::Pin<&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>> {std::pin::Pin::<&mut futures_async_stream::try_stream::from_generator::GenTryStream<[static generator@src/stream/src/executor/hash_join.rs:760:5: 760:61]>>::new_unchecked})) with 4052 obligations

It seems that Rust is very slow with resolving futures_async_stream. It took about 1sec for each resolving.

skyzh · 2022-10-17T00:42:39Z

To reproduce, RUSTC_LOG=debug ./risedev d

yuhao-su · 2022-10-17T02:33:03Z

Do you have time to look into why HashJoin is causing rustc to stuck?

I'll take a look.

skyzh · 2022-10-18T01:25:37Z

Unluckily, the bombing obligations seem to originate from StateTableIter

INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::ty::subst::GenericArg>: result=Ok(std::future::from_generator::GenFuture<[static generator@src/stream/src/common/mod.rs:31:45: 37:2]>) with 4052 obligations
INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::ty::subst::GenericArg>: result=Ok(std::future::from_generator::GenFuture<[static generator@risingwave_storage::table::streaming_table::state_table::StateTable<risingwave_storage::monitor::MonitoredStateStore<risingwave_storage::hummock::HummockStorage>>::iter::{closure#0}]>) with 4052 obligations
INFO rustc_trait_selection::traits::query::normalize normalize::<rustc_middle::ty::subst::GenericArg>: result=Ok(std::future::from_generator::GenFuture<[static generator@risingwave_storage::table::streaming_table::state_table::StateTable<risingwave_storage::monitor::MonitoredStateStore<risingwave_storage::hummock::HummockStorage>>::iter_with_pk_prefix::{closure#0}]>) with 4052 obligations

which cases all executors to have a lot of obligations, whereas hash join needs a lot of specialization, and cases the most problems.

skyzh · 2022-10-18T01:31:07Z

After changing iter_key_and_val to boxed, risingwave_stream compiles in 40 seconds (still seems slow). cc @BugenZhao now it's your turn to investigate what's happening 🥵

    pub async fn iter_key_and_val<'a>(
        &'a self,
        pk_prefix: &'a Row,
    ) -> StorageResult<RowStreamWithPk<'a, S>> {
        let (mem_table_iter, storage_iter_stream) = self
            .iter_with_pk_prefix_inner(pk_prefix, self.epoch())
            .await?;
        let storage_iter = storage_iter_stream.into_stream().boxed();

        Ok(
            StateTableRowIter::new(mem_table_iter, storage_iter, self.row_deserializer.clone())
                .into_stream().boxed(),
        )
    }

skyzh · 2022-10-18T01:32:23Z

My wild guess is that while rustc allows more lifetime generics in the latest version, some wrongly-labeled lifetimes will cause compile issues. Every time we add 'a at &self, there must be something going wrong.

skyzh · 2022-10-18T01:41:30Z

The issue now narrows down to risingwave_storage. It seems that it produces 4052 obligations...

Most of them are OutlivesPredicate (lifetime issues)

TennyZhuang · 2022-10-18T01:43:54Z

turn to @wenym1

TennyZhuang · 2022-10-18T01:44:57Z

IIRC storage have many unnecessary generics, but I’m not sure whether it’s related.

skyzh · 2022-10-18T02:27:03Z

I feel this issue more related to StateTable implementation than HummockStorage itself. The obligations will only bomb when processing StateTable-related code.

BugenZhao · 2022-10-18T02:29:22Z

Will implementing state table iterator manually with poll_next help?

skyzh · 2022-10-18T02:33:56Z

I'm not sure, but I'd suggest investigating where's the obligations from.

MrCroxx · 2022-10-19T01:40:52Z

Also the lifetime issue doesn't look like a false positive.
        let rotate_last_mut = |buffers: &'a mut Vec<_>| {
            buffers.push(DioBuffer::with_capacity_in(
                self.buffer_capacity,
                &DIO_BUFFER_ALLOCATOR,
            ));
            buffers.last_mut().unwrap()
        };
is defined for 'a mut Vec

where
rotate_last_mut(&mut self.buffers),
is following the lifetime of &mut self. buffers' lifetime has no relationship to 'a. The compiler's report seems correct.

cc @MrCroxx would you please help fix this?

Sure. I'll handle it.

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

lmatz · 2022-10-24T05:49:17Z

#1334 can be closed after this

skyzh · 2022-10-25T03:08:55Z

I'll take over this PR and get it merged with my pace.

victoryang00 · 2022-10-26T23:27:18Z

Thank you guys very much

skyzh · 2022-10-27T21:44:19Z

close in favor of #6025

build(toolchain): bump to 2022-09-05

3fe6787

github-actions bot added the type/build label Sep 5, 2022

xxchan mentioned this pull request Sep 5, 2022

type parameter out of range when substituting rust-lang/rust#101465

Closed

BugenZhao reviewed Sep 6, 2022

View reviewed changes

src/common/src/error.rs Show resolved Hide resolved

0913

4cbcd7a

xxchan changed the title ~~build(toolchain): bump to 2022-09-05~~ build(toolchain): bump to 2022-09-13 Sep 13, 2022

xxchan added 5 commits September 22, 2022 10:08

#![feature(let_chains)]

ffcf4b8

backtrace

a06b293

Merge remote-tracking branch 'origin/main' into xxchan/bump

095f8f3

20220922

e6fb0e4

clippy

124636d

xxchan changed the title ~~build(toolchain): bump to 2022-09-13~~ build(toolchain): bump to 2022-09-22 Sep 22, 2022

xxchan mentioned this pull request Sep 22, 2022

use cargo workspace inheritance #5507

Closed

xxchan and others added 2 commits September 22, 2022 20:18

Merge branch 'main' into xxchan/bump

e28ed62

Merge branch 'main' into xxchan/bump

158f264

xxchan added 3 commits October 17, 2022 00:03

20221016 -- it works!

e063818

Merge branch 'main' into xxchan/bump

c2b5692

fmt

f191ea1

xxchan marked this pull request as ready for review October 16, 2022 22:10

xxchan changed the title ~~build(toolchain): bump to 2022-09-22~~ build(toolchain): bump to 2022-10-16 Oct 16, 2022

xxchan commented Oct 16, 2022

View reviewed changes

src/storage/src/lib.rs Show resolved Hide resolved

BugenZhao approved these changes Oct 16, 2022

View reviewed changes

fix

69d18a9

skyzh approved these changes Oct 16, 2022

View reviewed changes

xxchan mentioned this pull request Oct 16, 2022

HummockError is too large #5869

Closed

xxchan mentioned this pull request Oct 17, 2022

style: fix warnings on latest clippy #5876

Merged

xxchan added 2 commits October 17, 2022 11:24

Merge branch 'main' into xxchan/bump

1261e7c

minor revert

2c5bc30

xxchan mentioned this pull request Oct 17, 2022

compilation stuck while resolving futures_async_stream rust-lang/rust#103147

Closed

BugenZhao added 3 commits October 22, 2022 22:40

chore(build): workaround for toolchain upgrade

db7e33f

Signed-off-by: Bugen Zhao <i@bugenzhao.com>

Merge branch 'bz2/prepare-toolchain-update' into xxchan/bump

6857085

fix

a0b2a11

skyzh force-pushed the xxchan/bump branch from 73db864 to a0b2a11 Compare October 23, 2022 03:05

This was referenced Oct 23, 2022

large number of obligation produced in async code causing compiler stuck rust-lang/rust#103423

Closed

Tracking: bump compiler toolchain to 1.65 #5977

Closed

BugenZhao mentioned this pull request Oct 24, 2022

chore(build): workaround for toolchain upgrade #5980

Merged

3 tasks

BugenZhao mentioned this pull request Oct 26, 2022

chore(build): bump toolchain to 2022-10-27 #6025

Merged

3 tasks

skyzh closed this Oct 27, 2022

MrCroxx mentioned this pull request Nov 3, 2022

chore: refine lifetime for rotate_last_mut #6193

Merged

3 tasks

xxchan deleted the xxchan/bump branch November 4, 2022 14:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build(toolchain): bump to 2022-10-16 #5119

build(toolchain): bump to 2022-10-16 #5119

xxchan commented Sep 5, 2022

xxchan commented Sep 13, 2022

BugenZhao commented Oct 8, 2022

BugenZhao Oct 16, 2022

BugenZhao Oct 16, 2022

xxchan Oct 16, 2022

skyzh commented Oct 17, 2022

skyzh commented Oct 17, 2022

skyzh commented Oct 17, 2022

yuhao-su commented Oct 17, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

TennyZhuang commented Oct 18, 2022

TennyZhuang commented Oct 18, 2022

skyzh commented Oct 18, 2022

BugenZhao commented Oct 18, 2022

skyzh commented Oct 18, 2022

MrCroxx commented Oct 19, 2022

lmatz commented Oct 24, 2022

skyzh commented Oct 25, 2022

victoryang00 commented Oct 26, 2022

skyzh commented Oct 27, 2022

		.ok_or(ValueEncodingError::InvalidNaiveTimeEncoding(secs, nano))?,
		.ok_or_else(\|\| ValueEncodingError::InvalidNaiveTimeEncoding(secs, nano))?,

build(toolchain): bump to 2022-10-16 #5119

build(toolchain): bump to 2022-10-16 #5119

Conversation

xxchan commented Sep 5, 2022

What's changed and what's your intention?

xxchan commented Sep 13, 2022

BugenZhao commented Oct 8, 2022

BugenZhao Oct 16, 2022

Choose a reason for hiding this comment

BugenZhao Oct 16, 2022

Choose a reason for hiding this comment

xxchan Oct 16, 2022

Choose a reason for hiding this comment

skyzh commented Oct 17, 2022

skyzh commented Oct 17, 2022

skyzh commented Oct 17, 2022

yuhao-su commented Oct 17, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

skyzh commented Oct 18, 2022

TennyZhuang commented Oct 18, 2022

TennyZhuang commented Oct 18, 2022

skyzh commented Oct 18, 2022

BugenZhao commented Oct 18, 2022

skyzh commented Oct 18, 2022

MrCroxx commented Oct 19, 2022

lmatz commented Oct 24, 2022

skyzh commented Oct 25, 2022

victoryang00 commented Oct 26, 2022

skyzh commented Oct 27, 2022